Automatic Diacritic Restoration for Resource-Scarce Languages
Identifieur interne : 000D63 ( Main/Exploration ); précédent : 000D62; suivant : 000D64Automatic Diacritic Restoration for Resource-Scarce Languages
Auteurs : Guy De Pauw [Belgique] ; W. Wagacha [Kenya] ; Gilles-Maurice De Schryver [Belgique, Afrique du Sud]Source :
- Lecture Notes in Computer Science [ 0302-9743 ] ; 2007.
Abstract
Abstract: The orthography of many resource-scarce languages includes diacritically marked characters. Falling outside the scope of the standard Latin encoding, these characters are often represented in digital language resources as their unmarked equivalents. This renders corpus compilation more difficult, as these languages typically do not have the benefit of large electronic dictionaries to perform diacritic restoration. This paper describes experiments with a machine learning approach that is able to automatically restore diacritics on the basis of local graphemic context. We apply the method to the African languages of Cilubà, Gĩkũyũ, Kĩkamba, Maa, Sesotho sa Leboa, Tshivenda and Yoruba and contrast it with experiments on Czech, Dutch, French, German and Romanian, as well as Vietnamese and Chinese Pinyin.
Url:
DOI: 10.1007/978-3-540-74628-7_24
Affiliations:
Links toward previous steps (curation, corpus...)
- to stream Istex, to step Corpus: 000F95
- to stream Istex, to step Curation: 000F46
- to stream Istex, to step Checkpoint: 000778
- to stream Main, to step Merge: 000D76
- to stream Main, to step Curation: 000D63
Le document en format XML
<record><TEI wicri:istexFullTextTei="biblStruct:series"><teiHeader><fileDesc><titleStmt><title xml:lang="en">Automatic Diacritic Restoration for Resource-Scarce Languages</title><author><name sortKey="De Pauw, Guy" sort="De Pauw, Guy" uniqKey="De Pauw G" first="Guy" last="De Pauw">Guy De Pauw</name></author><author><name sortKey="Wagacha, W" sort="Wagacha, W" uniqKey="Wagacha W" first="W." last="Wagacha">W. Wagacha</name></author><author><name sortKey="De Schryver, Gilles Maurice" sort="De Schryver, Gilles Maurice" uniqKey="De Schryver G" first="Gilles-Maurice" last="De Schryver">Gilles-Maurice De Schryver</name></author></titleStmt><publicationStmt><idno type="wicri:source">ISTEX</idno><idno type="RBID">ISTEX:66FA160FDA9F8DED4864CF1FC631F3ADF72BDB82</idno><date when="2007" year="2007">2007</date><idno type="doi">10.1007/978-3-540-74628-7_24</idno><idno type="url">https://api.istex.fr/document/66FA160FDA9F8DED4864CF1FC631F3ADF72BDB82/fulltext/pdf</idno><idno type="wicri:Area/Istex/Corpus">000F95</idno><idno type="wicri:Area/Istex/Curation">000F46</idno><idno type="wicri:Area/Istex/Checkpoint">000778</idno><idno type="wicri:doubleKey">0302-9743:2007:De Pauw G:automatic:diacritic:restoration</idno><idno type="wicri:Area/Main/Merge">000D76</idno><idno type="wicri:Area/Main/Curation">000D63</idno><idno type="wicri:Area/Main/Exploration">000D63</idno></publicationStmt><sourceDesc><biblStruct><analytic><title level="a" type="main" xml:lang="en">Automatic Diacritic Restoration for Resource-Scarce Languages</title><author><name sortKey="De Pauw, Guy" sort="De Pauw, Guy" uniqKey="De Pauw G" first="Guy" last="De Pauw">Guy De Pauw</name><affiliation wicri:level="4"><country xml:lang="fr">Belgique</country><wicri:regionArea>CNTS - Language Technology Group, University of Antwerp</wicri:regionArea><placeName><settlement type="city">Anvers</settlement><region type="district" nuts="2">Province d'Anvers</region></placeName><orgName type="university">Université d'Anvers</orgName></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Belgique</country></affiliation></author><author><name sortKey="Wagacha, W" sort="Wagacha, W" uniqKey="Wagacha W" first="W." last="Wagacha">W. Wagacha</name><affiliation wicri:level="1"><country xml:lang="fr">Kenya</country><wicri:regionArea>School of Computing and Informatics, University of Nairobi</wicri:regionArea><wicri:noRegion>University of Nairobi</wicri:noRegion></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Kenya</country></affiliation></author><author><name sortKey="De Schryver, Gilles Maurice" sort="De Schryver, Gilles Maurice" uniqKey="De Schryver G" first="Gilles-Maurice" last="De Schryver">Gilles-Maurice De Schryver</name><affiliation wicri:level="1"><country xml:lang="fr">Belgique</country><wicri:regionArea>African Languages and Cultures, Ghent University</wicri:regionArea><wicri:noRegion>Ghent University</wicri:noRegion></affiliation><affiliation wicri:level="1"><country xml:lang="fr">Afrique du Sud</country><wicri:regionArea>Xhosa Department, University of the Western Cape</wicri:regionArea><wicri:noRegion>University of the Western Cape</wicri:noRegion></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Belgique</country></affiliation></author></analytic><monogr></monogr><series><title level="s">Lecture Notes in Computer Science</title><imprint><date>2007</date></imprint><idno type="ISSN">0302-9743</idno><idno type="eISSN">1611-3349</idno><idno type="ISSN">0302-9743</idno></series><idno type="istex">66FA160FDA9F8DED4864CF1FC631F3ADF72BDB82</idno><idno type="DOI">10.1007/978-3-540-74628-7_24</idno><idno type="ChapterID">24</idno><idno type="ChapterID">Chap24</idno></biblStruct></sourceDesc><seriesStmt><idno type="ISSN">0302-9743</idno></seriesStmt></fileDesc><profileDesc><textClass></textClass><langUsage><language ident="en">en</language></langUsage></profileDesc></teiHeader><front><div type="abstract" xml:lang="en">Abstract: The orthography of many resource-scarce languages includes diacritically marked characters. Falling outside the scope of the standard Latin encoding, these characters are often represented in digital language resources as their unmarked equivalents. This renders corpus compilation more difficult, as these languages typically do not have the benefit of large electronic dictionaries to perform diacritic restoration. This paper describes experiments with a machine learning approach that is able to automatically restore diacritics on the basis of local graphemic context. We apply the method to the African languages of Cilubà, Gĩkũyũ, Kĩkamba, Maa, Sesotho sa Leboa, Tshivenda and Yoruba and contrast it with experiments on Czech, Dutch, French, German and Romanian, as well as Vietnamese and Chinese Pinyin.</div></front></TEI><affiliations><list><country><li>Afrique du Sud</li><li>Belgique</li><li>Kenya</li></country><region><li>Province d'Anvers</li></region><settlement><li>Anvers</li></settlement><orgName><li>Université d'Anvers</li></orgName></list><tree><country name="Belgique"><region name="Province d'Anvers"><name sortKey="De Pauw, Guy" sort="De Pauw, Guy" uniqKey="De Pauw G" first="Guy" last="De Pauw">Guy De Pauw</name></region><name sortKey="De Pauw, Guy" sort="De Pauw, Guy" uniqKey="De Pauw G" first="Guy" last="De Pauw">Guy De Pauw</name><name sortKey="De Schryver, Gilles Maurice" sort="De Schryver, Gilles Maurice" uniqKey="De Schryver G" first="Gilles-Maurice" last="De Schryver">Gilles-Maurice De Schryver</name><name sortKey="De Schryver, Gilles Maurice" sort="De Schryver, Gilles Maurice" uniqKey="De Schryver G" first="Gilles-Maurice" last="De Schryver">Gilles-Maurice De Schryver</name></country><country name="Kenya"><noRegion><name sortKey="Wagacha, W" sort="Wagacha, W" uniqKey="Wagacha W" first="W." last="Wagacha">W. Wagacha</name></noRegion><name sortKey="Wagacha, W" sort="Wagacha, W" uniqKey="Wagacha W" first="W." last="Wagacha">W. Wagacha</name></country><country name="Afrique du Sud"><noRegion><name sortKey="De Schryver, Gilles Maurice" sort="De Schryver, Gilles Maurice" uniqKey="De Schryver G" first="Gilles-Maurice" last="De Schryver">Gilles-Maurice De Schryver</name></noRegion></country></tree></affiliations></record>Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Ticri/CIDE/explor/OcrV1/Data/Main/Exploration
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 000D63 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Main/Exploration/biblio.hfd -nk 000D63 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien
|wiki= Ticri/CIDE
|area= OcrV1
|flux= Main
|étape= Exploration
|type= RBID
|clé= ISTEX:66FA160FDA9F8DED4864CF1FC631F3ADF72BDB82
|texte= Automatic Diacritic Restoration for Resource-Scarce Languages
}}
| This area was generated with Dilib version V0.6.32. |  |